Software application developers generally conduct extensive testing of their products to ensure that they work as desired. The products often need to be tested using various operating systems and various hardware components, often located at various physical or geographical sites.
As the computer field continues to grow at a rapid pace, the need to test application performance also continues to grow. Applications are continually being developed, and the ability to test these new applications greatly facilitates the process of application development. Thus, the application testing field has grown and continues to grow to keep up with the rapid expansion of the software field.
Typically, the software product development process (performed by a “programmer”) is a separate process from the software product testing process (performed by a “tester”). A programmer will often have a good understanding of how a software product will behave under everyday processing conditions; however, the programmer may not have completely considered how the software product will behave under rare or unexpected conditions. Often this can be the cause of errors within the software product. Additionally, most software products are developed by keyboarding characters into a computer; thus, the possibility of typographical errors always exists, and such errors may be randomly distributed throughout the software product. As a result, an error may be located in an obscure logical path, and thus be difficult to find.
As a result of the potential for errors during the development of a software product, in-depth testing is necessary to identify all such errors. The testing process usually consists of having a tester design a test to supply predetermined variable inputs to the program and collect the resulting output. This output is then analyzed by comparing it to the desired or expected output for a particular set of input variables.
A tester designing a test for a particular application typically first ascertains the specifications of the software product being tested. These specifications are a basic “blueprint” used by the programmer when designing the product, and by providing them to the tester, the tester can better understand how the product is intended to function. From the product specifications, the tester creates a number of “assertions.” An assertion is a determination of how the product will behave when it is performing properly in a given situation or in response to a given set of input values.
For each assertion, the tester then develops a set of software “test cases.” Test cases are scenarios which provide various situations to the product, as described in more detail below. The goal of the test cases is to verify the assertions. Test cases can be short operations which verify one particular function, or they can be longer operations which verify entire processes. For example, an assertion could be made regarding how the program will respond when the installation function is executed within a given operating system or platform. Thus, an entire installation procedure might comprise a single test case. Various test cases are created by the tester in an effort to fully test all aspects of the product.
If every assertion for a given product operating in a particular operating system is verified by using a test case, the product is deemed to be fully tested. Typically, a large number of test cases are required to be developed to test the product in conjunction with various combinations of software and hardware platforms.
Often the tester is provided with little information regarding the structure and design of the code comprising the software product. The product is tested by applying test cases, observing the output of the software product, and comparing the output to the expected result. This type of testing is referred to as “black box” testing. These tests are often conducted manually, and require the tester to carefully record the sequence of test case inputs and the output results from the product.
Separate test cases are typically developed for every product and every platform in which the product could be used. Many scenarios for which test cases are developed are appropriate tests for all products and all platforms. For example, a test case simulating the software installation process is a test case which is desired for virtually every product. In addition, a separate test case to test the installation process for each platform in which the product operates (e.g., windows, MS-DOS, UNIX, etc.) is also appropriate.
In an effort to automate the testing process, scripting and macro languages are often used to conduct the testing process. A script tool or macro tool allows some degree of automation in the test sequence, and aids in reproducing test cases. Examples of such tools are “.BAT” files in the MS-DOS environment and “shell scripts” in the UNIX environment. However, while these tools have aided in the automation of the testing input, collection and analysis of the test results remains largely a manual process.
An additional problem with using scripting/macro tools in testing is that the tool itself and the test cases being processed using the tool must be configured to each computing environment in which the software product is to be tested. For example, a particular operation in a Microsoft Windows® environment may be tested by simulating an “OPEN” menu operation, while a similar function may require an “OK” menu operation in an Apple Macintosh® environment. The test scripts must be changed to test the same function of an application operating in two different platforms.
The goal of “black box” testing is to exercise all logical possibilities for a given product and assure that the product works as desired in every case, and often for several different platforms. However, in general, it is difficult to exhaustively test a computer program. The number of possibilities that need to be tested can be extensive, and complete exhaustive testing can take more time than is practicable.
The need for the development of a large number of specific test cases has made the testing of software products very time consuming and costly. As a result, a need exists for an automated test system that has the ability to apply test cases to a large number of products operating in a variety of platforms. Furthermore, it is desired to have a test system that can automatically collect, store and analyze the results and provide the test engineer with appropriate, meaningful feedback on the performance of the product.